Introducing Per-Document Embedding Fine-Tuning

Vector search inevitably involves representing the documents or items via embeddings at indexing time. In most cases, these embeddings remain frozen and contain strictly the information that the embedding model was able to deduct from the inputs.

In many cases, it would be useful to be able to update the embeddings when new information about the document or item arrives. Until now, this was only possible by fine-tuning the embedding model and re-embedding the entire collection of documents, a long, costly, and risky process.

We introduce per-document fine-tuning: using an embedding model with deterministic dimensions, we were able to fine-tune individual embeddings with new information while the model remained frozen. In the case of multimodal embeddings, this even enabled us to update an image embedding with additional information not found in the image. For example an e-commerce product whose embedding was generated from its picture (i. e. wide-legged diamond studded jeans) can have its embedding augmented with abstract characteristics such as the material (organic cotton), the esthetic (hippie look), and use cases (festival wear).

We will demonstrate the effect on precision and recall of enhancing multimodal e-commerce product embeddings with abstract characteristics using this method.

Philippe Bouzaglou

Vectra Multimodal Embeddings

Philippe pioneered the concept of the Social Graph while attending Harvard with Mark Zuckerberg. He has since gone on to achieve significant contributions in the fields of data science and machine learning. Recently, he was VP of Data Science and Machine Learning and responsible for search at Gorillas, the quick-commerce company that pioneered grocery delivery within the hour. He is now the technical founder at Vectra, an AI company creating the foundation model for e-commerce search.